已经证明了对比学习适合学习句子嵌入，可以显着提高语义文本相似性（STS）任务。最近，大型对比学习模型，例如句子T5倾向于学到更强大的句子嵌入。虽然有效，但由于计算资源或时间成本限制，这种大型型号很难在线服务。为了解决这个问题，通常采用知识蒸馏（KD），这可以将大型“教师”模型压缩成一个小的“学生”模型，但通常会遭受一些性能损失。在这里，我们提出了一个增强的KD框架，称为蒸馏 - 对比度（迪斯科）。所提出的迪斯科框架首先利用KD将大句子嵌入模型的能力转移到大型未标记数据的小学生模型，然后在标记的训练数据上具有对比学习的学生模型。对于迪斯科舞厅的KD进程，我们进一步提出了对比的知识蒸馏（CKD），以增强教师模型培训，KD和学生模型的一致性，这可能会提高迅速学习的表现。 7 STS基准测试的广泛实验表明，使用所提出的迪斯科和CKD培训的学生模型很少或甚至没有性能损失，并且始终如一地优于相同参数大小的相应对应物。令人惊讶的是，我们的110米学生模型甚至可以优于最新的最新（SOTA）模型，即句子T5（11B），只有1％的参数。

对比学习一直吸引着学习无监督的句子嵌入。当前的最新无监督方法是无监督的SIMCSE（UNSUP-SIMCSE）。 Unsup-Simcse将辍学作为最小数据增强方法，并将相同的输入句子传递给预训练的变压器编码器（带有掉落的掉落）两次，以获取两个相应的嵌入式以构建正对。由于句子的长度信息通常会由于使用嵌入变压器中的位置嵌入而编码到句子嵌入中，因此Unsup-Simcse中的每个正对实际上包含相同的长度信息。因此，接受这些正面对训练的Unsup-Simcse可能是有偏见的，这往往会考虑到语义上相同长度或相似长度的句子更相似。通过统计观察，我们发现Unsup-Simcse确实存在这样的问题。为了减轻它，我们应用了一个简单的重复操作来修改输入句子，然后分别将输入句子及其修改后的对应物传递给预训练的变压器编码器，以获取阳性对。此外，我们从计算机视觉社区中汲取灵感，并引入动量对比度，从而扩大了负面对的数量，而没有其他计算。提出的两种修改分别应用于正和负对，并构建一种新的句子嵌入方法，称为增强的Unsup-Simcse（ESIMCSE）。我们在几个基准数据集W.R.T上评估了所提出的ESIMCSE，语义文本相似性（STS）任务。实验结果表明，ESIMCSE的表现优于最先进的undup-Simcse，而Bert基碱的平均长矛相关性为2.02％。

Several self-supervised representation learning methods have been proposed for reinforcement learning (RL) with rich observations. For real-world applications of RL, recovering underlying latent states is crucial, particularly when sensory inputs contain irrelevant and exogenous information. In this work, we study how information bottlenecks can be used to construct latent states efficiently in the presence of task-irrelevant information. We propose architectures that utilize variational and discrete information bottlenecks, coined as RepDIB, to learn structured factorized representations. Exploiting the expressiveness bought by factorized representations, we introduce a simple, yet effective, bottleneck that can be integrated with any existing self-supervised objective for RL. We demonstrate this across several online and offline RL benchmarks, along with a real robot arm task, where we find that compressed representations with RepDIB can lead to strong performance improvements, as the learned bottlenecks help predict only the relevant state while ignoring irrelevant information.

Deep learning classifiers provide the most accurate means of automatically diagnosing diabetic retinopathy (DR) based on optical coherence tomography (OCT) and its angiography (OCTA). The power of these models is attributable in part to the inclusion of hidden layers that provide the complexity required to achieve a desired task. However, hidden layers also render algorithm outputs difficult to interpret. Here we introduce a novel biomarker activation map (BAM) framework based on generative adversarial learning that allows clinicians to verify and understand classifiers decision-making. A data set including 456 macular scans were graded as non-referable or referable DR based on current clinical standards. A DR classifier that was used to evaluate our BAM was first trained based on this data set. The BAM generation framework was designed by combing two U-shaped generators to provide meaningful interpretability to this classifier. The main generator was trained to take referable scans as input and produce an output that would be classified by the classifier as non-referable. The BAM is then constructed as the difference image between the output and input of the main generator. To ensure that the BAM only highlights classifier-utilized biomarkers an assistant generator was trained to do the opposite, producing scans that would be classified as referable by the classifier from non-referable scans. The generated BAMs highlighted known pathologic features including nonperfusion area and retinal fluid. A fully interpretable classifier based on these highlights could help clinicians better utilize and verify automated DR diagnosis.

LiDAR-based 3D Object detectors have achieved impressive performances in many benchmarks, however, multisensors fusion-based techniques are promising to further improve the results. PointPainting, as a recently proposed framework, can add the semantic information from the 2D image into the 3D LiDAR point by the painting operation to boost the detection performance. However, due to the limited resolution of 2D feature maps, severe boundary-blurring effect happens during re-projection of 2D semantic segmentation into the 3D point clouds. To well handle this limitation, a general multimodal fusion framework MSF has been proposed to fuse the semantic information from both the 2D image and 3D points scene parsing results. Specifically, MSF includes three main modules. First, SOTA off-the-shelf 2D/3D semantic segmentation approaches are employed to generate the parsing results for 2D images and 3D point clouds. The 2D semantic information is further re-projected into the 3D point clouds with calibrated parameters. To handle the misalignment between the 2D and 3D parsing results, an AAF module is proposed to fuse them by learning an adaptive fusion score. Then the point cloud with the fused semantic label is sent to the following 3D object detectors. Furthermore, we propose a DFF module to aggregate deep features in different levels to boost the final detection performance. The effectiveness of the framework has been verified on two public large-scale 3D object detection benchmarks by comparing with different baselines. The experimental results show that the proposed fusion strategies can significantly improve the detection performance compared to the methods using only point clouds and the methods using only 2D semantic information. Most importantly, the proposed approach significantly outperforms other approaches and sets new SOTA results on the nuScenes testing benchmark.

Safety-critical Autonomous Systems require trustworthy and transparent decision-making process to be deployable in the real world. The advancement of Machine Learning introduces high performance but largely through black-box algorithms. We focus the discussion of explainability specifically with Autonomous Vehicles (AVs). As a safety-critical system, AVs provide the unique opportunity to utilize cutting-edge Machine Learning techniques while requiring transparency in decision making. Interpretability in every action the AV takes becomes crucial in post-hoc analysis where blame assignment might be necessary. In this paper, we provide positioning on how researchers could consider incorporating explainability and interpretability into design and optimization of separate Autonomous Vehicle modules including Perception, Planning, and Control.

Graphic sketch representations are effective for representing sketches. Existing methods take the patches cropped from sketches as the graph nodes, and construct the edges based on sketch's drawing order or Euclidean distances on the canvas. However, the drawing order of a sketch may not be unique, while the patches from semantically related parts of a sketch may be far away from each other on the canvas. In this paper, we propose an order-invariant, semantics-aware method for graphic sketch representations. The cropped sketch patches are linked according to their global semantics or local geometric shapes, namely the synonymous proximity, by computing the cosine similarity between the captured patch embeddings. Such constructed edges are learnable to adapt to the variation of sketch drawings, which enable the message passing among synonymous patches. Aggregating the messages from synonymous patches by graph convolutional networks plays a role of denoising, which is beneficial to produce robust patch embeddings and accurate sketch representations. Furthermore, we enforce a clustering constraint over the embeddings jointly with the network learning. The synonymous patches are self-organized as compact clusters, and their embeddings are guided to move towards their assigned cluster centroids. It raises the accuracy of the computed synonymous proximity. Experimental results show that our method significantly improves the performance on both controllable sketch synthesis and sketch healing.

Unsupervised domain adaptation (UDA) has been highly successful in transferring knowledge acquired from a label-rich source domain to a label-scarce target domain. Open-set domain adaptation (ODA) and universal domain adaptation (UNDA) have been proposed as solutions to the problem concerning the presence of additional novel categories in the target domain. Existing ODA and UNDA approaches treat all novel categories as one unified unknown class and attempt to detect this unknown class during the training process. We find that domain variance leads to more significant view-noise in unsupervised data augmentation, affecting the further applications of contrastive learning~(CL), as well as the current closed-set classifier and open-set classifier causing the model to be overconfident in novel class discovery. To address the above two issues, we propose Soft-contrastive All-in-one Network~(SAN) for ODA and UNDA tasks. SAN includes a novel data-augmentation-based CL loss, which is used to improve the representational capability, and a more human-intuitive classifier, which is used to improve the new class discovery capability. The soft contrastive learning~(SCL) loss is used to weaken the adverse effects of the data-augmentation label noise problem, which is amplified in domain transfer. The All-in-One~(AIO) classifier overcomes the overconfidence problem of the current mainstream closed-set classifier and open-set classifier in a more human-intuitive way. The visualization results and ablation experiments demonstrate the importance of the two proposed innovations. Moreover, extensive experimental results on ODA and UNDA show that SAN has advantages over the existing state-of-the-art methods.

The miniaturization and mobility of computer vision systems are limited by the heavy computational burden and the size of optical lenses. Here, we propose to use a ultra-thin diffractive optical element to implement passive optical convolution. A division adjoint opto-electronic co-design method is also proposed. In our simulation experiments, the first few convolutional layers of the neural network can be replaced by optical convolution in a classification task on the CIFAR-10 dataset with no power consumption, while similar performance can be obtained.

神经辐射场（NERF）已成功用于场景表示。最近的工作还使用基于NERF的环境表示形式开发了机器人导航和操纵系统。由于对象定位是许多机器人应用的基础，因此进一步释放了机器人系统中NERF的潜力，我们研究了NERF场景中的对象定位。我们提出了一个基于变压器的框架NERF-LOC，以在NERF场景中提取3D边界对象框。 Nerf-Loc将预先训练的NERF模型和相机视图作为输入，并产生标记为3D边界对象的框作为输出。具体来说，我们设计了一对平行的变压器编码器分支，即粗流和细流，以编码目标对象的上下文和详细信息。然后将编码的功能与注意层融合在一起，以减轻准确对象定位的歧义。我们已经将我们的方法与基于传统变压器的方法进行了比较，我们的方法可以实现更好的性能。此外，我们还提出了第一个基于NERF样品的对象定位基准Nerflocbench。